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INTRODUCTION 




What is the 2003 NAAL? 

Sponsored by the National Center for Education Statistics (NCES) in the U.S. Department of 
Education's Institute of Education Sciences, the 2003 National Assessment of Adult Literacy (NAAL) is 
a nationally representative assessment of literacy among adults (age 16 and older) residing in 
households and prisons in the United States. It is the first assessment of the nation's progress in adult 
literacy since the 1992 National Adult Literacy Survey (NALS). 

What do the results cover? 

Results from the 2003 NAAL cover the status and progress of literacy in the nation, the literacy skill 
levels of American adults (including the least-literate adults), various factors associated with literacy, 
and the application of literacy skills to health-related materials. NAAL also will provide the results of 
state-level assessments for six participating states and a national study on literacy among the 
prison population. 

The first results from the 2003 NAAL appear in A First Look at the Literacy of America's Adults in the 21st 
Century (Kutner, Greenberg, and Baer 2005). Later reports will provide additional results from and 
information about the assessment. 

What is the purpose of this publication? 

The 2003 NAAL is a complex assessment with several components and various types of data. The 
primary purpose of this publication is to describe the assessment's key features and data types.Thus, 
the publication covers the critical concepts and features carried over from the 1 992 assessment, as 
well as those new to the 2003 assessment — for example, new performance levels that are used to 
report results (see section 2) and new components that provide expanded data on the least-literate 
adults and on the role of basic skills in adult literacy performance (see section 3). By providing an 
overall picture of important goals and aspects of the 2003 NAAL, the publication provides a context 
for interpreting the results. 
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VIEWING ADULT LITERACY 
FROM A FUNCTIONAL PERSPECTIVE 




NAAL measures how well U.S. adults perform tasks with printed materials 

As a part of their everyday lives, adults in the United States interact with a variety of printed and other 
written materials to perform a multitude of tasks. A comprehensive list of such tasks would be 
virtuallyendless.lt would include such activities as balancing a checkbook, following directions on a 
prescription medicine bottle, filling out a job application, consulting a bus schedule, correctly 
interpreting a chart in the newspaper, and using written instructions to operate a voting machine. 

The National Assessment of Adult Literacy (NAAL) measures the ability of a nationally representative 
sample of adults to perform literacy tasks similar to those that they encounter in their daily lives. 
Statistical procedures ensure that NAAL participants represent the entire population of U.S. adults who 
are age 16 and older and live in households or prisons. In 2003, the 19,714 adults who participated in 
NAAL represented a U.S. adult population of about 222 million. (This population estimate was 
calculated by NAAL researchers based on data from the U.S. Census Bureau's 2003 Current Population 
Survey and the Bureau of Justice Statistics' midyear 2003 National Prisoner Statistics.) 

Like other adults, NAAL participants bring to literacy tasks a full range of backgrounds, experiences, 
and skill levels. Like real-life tasks, NAAL tasks vary with respect to the difficulty of the materials used 
as well as the complexity of the actions to be performed. However, in order to be fair to all 
participants, none of the tasks require specialized background knowledge, and all of them were 
reviewed for bias against particular groups. 

Adults need literacy skills in order to function 

Literacy is not a single skill or quality that one either possesses or lacks. Rather, it encompasses 
various types of skills that different individuals possess to varying degrees. There are different levels 
and types of literacy, which reflect the ability to perform a wide variety of tasks using written 
materials that differ in nature and complexity. A common thread across all literacy tasks is that each 
has a purpose — whether that purpose is to pay the telephone bill or to understand a piece of poetry. 
All U.S. adults must successfully perform literacy tasks in order to adequately function — that is, to 
meet personal and employment goals as well as contribute to the community. 

NAAL tasks reflect a definition of literacy that emphasizes the use of written materials to function 
adequately in one's environment and to develop as an individual. Of course, the actual literacy 
tasks that individuals must perform in their daily lives vary to some extent depending on the 
nature of their work and personal goals. However, virtually all literacy tasks require certain 
underlying skills, such as the ability to read and understand common words. NAAL measures 
adults' performance on a range of tasks mimicking actual tasks encountered by adults in the 
United States. Adults with very low levels of performance on NAAL tasks may be unable to 
function adequately in 21st century America. 




NAAL examines three literacy areas — prose, document, and quantitative 

NAAL reports a separate score for each of three literacy areas: 

Prose literacy refers to the knowledge and skills needed to perform prose tasks — that is, to search, 
comprehend, and use continuous texts. Prose examples include editorials, news stories, brochures, 
and instructional materials. 

Document literacy refers to the knowledge and skills needed to perform document tasks — that is, 
to search, comprehend, and use noncontinuous texts in various formats. Document examples 
include job applications, payroll forms, transportation schedules, maps, tables, and drug or food 
labels. 

Quantitative literacy refers to the knowledge and skills required to perform quantitative tasks — that 
is, to identify and perform computations, either alone or sequentially, using numbers embedded in 
printed materials. Examples include balancing a checkbook, computing a tip, completing an order 
form, or determining the amount of interest on a loan from an advertisement. 

The Framework for the 2003 National Assessment of Adult Literacy (White and McCloskey forthcoming) 
discusses the three literacy areas in detail. Underlying the prose, document, and quantitative tasks is NAAL's 
task-based definition of literacy (figure 1 ). 



Figure 1 . Task-based definition of literacy 



Literacy is the ability to use printed and written information to function in society, to achieve 
one's goals, and to develop one's knowledge and potential. 




NAAL's task-based definition of 
literacy implies the ability to 
meet literacy demands at home, 
in the workplace, and in the 
community by performing prose, 
document, and quantitative 
tasks. 



SOURCE: White, S., and McCloskey, M. (forthcoming). Framework for the 2003 National Assessment of Adult 
Literacy (NCES 2006-473). U.S. Department of Education. Washington, DC: National Center for Education 
Statistics. 



Sample assessment tasks have been released to the public 

To provide a clearer picture of the types of tasks that NAAL participants are asked to perform, the 
National Center for Education Statistics (NCES) has released numerous assessment tasks (i.e., test 
questions) that either were used in the 2003 NAAL or are similar to those used in the 2003 NAAL. 
Most of these tasks were used in the 1992 National Adult Literacy Survey (NALS).from which NAAL 
evolved. Below are a few examples. Almost 100 tasks are currently available on the NAAL website 
( http://nces.ed.aov/naalATestOuestions.asp ). NCES plans to make more assessment tasks available in 
the future, including many of those used in 2003. However, not all of the tasks used in the 2003 
administration of NAAL can be made public, because some of them will be reused in future 
administrations to allow comparisons across time. 

Figure 2 shows an easy prose task (performed correctly by 83 percent of adults). This task requires 
participants to search a short text to locate a single piece of easily identifiable information. In more 
difficult prose tasks, the requirements include making inferences, comparing and contrasting 
information, and synthesizing pieces of information from long and complex passages. 



Figure 2. Example of an easy prose task 



Underline the sentence that tells how the Social Security 
Administration defines the term "blind." 



WHAT IS SSI? 

SSI stands for supplemental security income. It is a Federal program 
run by the Social Security Administration. It pays monthly checks to 
aged, blind, and disabled people who do not have much income or 
resources. 

Under SSI, aged means you are 65 or older. Blind means the vision 
in your better eye is 20/200 or less or you have a limited visual field 
of 20 degrees or less. Disabled means you have a severe physical or 
mental condition that keeps you from doing any substantial gainful 
work, and medical evidence shows it is expected to last at least 12 
months or result in death. 




SOURCE: U.S. Department of Education, National Center for Education Statistics, 1992 National Adult 
Literacy Survey (NALS). Available in the Sample Questions section of the NAAL website ( http://nces.ed.gov/ 
naal/TestOuestions.asp) . 




Figure 3 shows a moderately difficult document task (performed correctly by 56 percent of adults). 
This task requires participants to determine which type of sandpaper to buy for a specific job. To do 
this, they need to identify the correct row, column, and cell in a complex table that contains 
subcategories. To select the correct row, participants must find the "WOOD" category and the 
"Preparation for Sealing"subcategory in the list at the far left of the table.To select the correct column 
and cell, they must first identify "GARNET" as the main column heading that is relevant, then follow 
the row they selected to the shaded cell under this main heading. Finally, they must connect the 
abbreviation “F" in the subordinate column heading with the word "Fine" in the key below the table. 

More difficult document tasks have requirements such as comparing, contrasting, and drawing high- 
level inferences from multiple pieces of information embedded in complex documents. At the other 
end of the spectrum, the simplest document tasks require only actions such as signing a form in the 
right place or appropriately filling in blanks. 



Figure 3. Example of a moderately difficult document task 



You need to smooth wood in preparation for sealing and plan to buy garnet 
sandpaper. What type of sandpaper should you buy? 




ABRASIVE SELECTION GUIDE 

PRom irrnnisi® 

MATERIAL & OPERATION 

WOOD 
Paint Removal 
Heavy Stock Removal 
Moderate Stock Removal 
Preparation for Sealing 
After Sealer 
Between Coats 
After Final Coat 
METAL 

Rust and Paint Removal 
Light Stock Removal 
Preparation for Priming 
Finishing and Polishing 
After Primer 
Between Coats 
After Final Coat 
PLASTIC & FIBERGLASS 
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Light Stock Removal 
Finishing & Scuffing 

EC = Extra Coarse C = Coarse M = Medium F = Fine VF = Very Fine EF = Extra Fine SF = Super Fine UF = Ultra Fine 



SAFETY INFORMATION: 

■ Wear approved safety goggles 
when sanding. 



■ Use particle/dust mask or other ■ When using power tools, follow 
means to prevent inhalation of manufacturer's recommended 
sanding dust. procedures and safety instructions. 



NOTE: Reprinted by permission of and copyrighted by the 3M Company. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 1 992 National Adult Literacy 
Survey (NALS). Available in the Sample Questions section of the NAAL website (http://nces.ed.gov/naal/ 
TestQuestions.asp) . 




Figure 4 shows one of the more difficult quantitative tasks (performed correctly by 29 percent of 
adults). This task requires using information on an automobile maintenance record to compute the 
gas mileage since the previous fill-up. To perform this task, participants must determine which 
numbers on the maintenance record are relevant to the task and what specific computations must 
be performed to get the answer. The appropriate steps are to subtract the mileage on March 2 
(42,775) from the mileage on March 9 (43,083), then divide the result (308 miles) by the number of 
gallons used (1 2.5). If participants perform these computation steps correctly, they will find that the 
car got about 25 miles per gallon since it was filled with gas on March 2. If participants get any of the 
computation steps wrong, however, they will not obtain the correct answer. A simpler task might 
involve solving a single equation using only numbers that actually appear in the document. 



Figure 4. Example of a difficult quantitative task 



On March 9 you filled your car with gas. Calculate how many miles 
per gallon your car got since you filled it up with gas on March 2. 
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SOURCE: U.S. Department of Education, National Center for Education Statistics, 1 992 National Adult Literacy 
Survey (NALS). Available in the Sample Questions section of the NAAL website ( http://nces.ed.gov/ 
naal/TestOuestions.asp ). 




NAAL includes a new measure of health literacy 

The U.S. Department of Health and Human Services (HHS) has adopted the following definition of 
health literacy: "The degree to which individuals have the capacity to obtain, process, and 
understand basic health information and services needed to make appropriate health decisions" 
(Selden et al. 2000, cited in HHS 2000). Although health literacy involves factors that NAAL cannot 
measure — such as the ability to communicate orally — NAAL will provide a portrait of one important 
aspect of the health literacy of the U.S. adult population. 

NAAL's health literacy component is the first-ever national assessment designed specifically to 
measure adults' ability to use literacy skills to read and understand health-related information. 
Health-related materials used in the assessment include medication information, medical 
instructions, health insurance forms, and prevention and wellness information. Nearly one-fifth of the 
NAAL tasks are health related, and all participants perform some health-related tasks. One example 
appears in figure 5. 

Each health-related task is also classified as a prose, document, or quantitative task. Thus, health- 
related tasks are included with other tasks when calculating the prose, document, and quantitative 
literacy scores. However, NAAL also includes a separate health literacy score, based solely on the 
health-related tasks. This score will measure the ability of adults with various demographic and 
background characteristics to effectively use health-related information, provide input for the 
development of health-related information and programs for these adults, and establish a baseline 
for tracking progress in future assessments. 



Figure 5. Example of a health-related task 



Copy three food sources, named in the almanac, that contain 
vitamin E. 



Vitamin E (tocopherol) — helps protect red blood cells. 
May aid the circulatory system and counteract the aging 
process. Best sources: wheat germ, whole grains, eggs, 
peanuts, organ meats, margarine, vegetable oils, green leafy 
vegetables. 




SOURCE: U.S. Department of Education, National Center for Education Statistics, 1 992 National Adult Literacy 
Survey (NALS). Available in the Sample Questions section of the NAAL website ( http://nces.ed.gov/ 
naal/TestOuestions.asp ). 




COMPARING ADULT LITERACY 
IN 1992 AND 2003 



NAAL measures the three types of literacy that were measured in 1992 

One important goal of the 2003 National Assessment of Adult Literacy (NAAL) is to provide 
information on changes in adult literacy performance since 1992. Accordingly, the 2003 NAAL 
provides scores for the same three literacy areas — prose, document, and quantitative — that were 
examined in the 1 992 National Adult Literacy Survey (NALS). In order to provide trend data on adult 
literacy in the future, the National Center for Education Statistics (NCES) plans to conduct 
assessments of adult literacy periodically. 



Cross-year comparisons are available for the nation, one state, and prisons 

Changes over time in the literacy of adults living in the United States are of interest to diverse 
audiences, ranging from the general public to policymakers. For example, policymakers may use 
information about literacy changes to justify the creation or improvement of literacy programs, to 
provide grants for further research, or for accountability purposes. Libraries may use this information 
to help ensure that their policies and materials are appropriate for a broad spectrum of adults. 
Education researchers may explore possible causes of literacy changes and possible methods for 
promoting higher levels of literacy among adults. Members of the general public may be interested 
to know whether recent changes in the nation's demographic profile have been accompanied by 
changes in adult literacy. 

In addition to results for the nation, both the 1992 NALS and the 2003 NAAL provide the results of 
state-level assessments for those states that chose to participate in state assessments. Participating 
states can compare their state with other participating states and with the nation. In 2003, the 
following states participated in state assessments: Kentucky, Maryland, Massachusetts, Missouri, New 
York, and Oklahoma. State-level results can help states determine where to target investment in adult 
education, training programs, and other services. In addition, the results establish a baseline for future 
assessments. Because New York participated in 1992 as well as in 2003, the 2003 results for New York 
include information on state-level changes across time. 








Like the 1992 NALS, the 2003 NAAL includes an assessment of literacy among the prison population. 
This population includes only adults in state and federal prisons, not those in local jails or other types 
of institutions. (Sampled adults living in households but temporarily in local jails — where the median 
stay is about 2 weeks — were considered part of the household population, not the prison 
population. Since the data collection period was about 10 months long, these adults were 
interviewed in their homes when they got out of jail.) In both assessment years, the prison sample 
was representative of the prison population at the national level, allowing separate literacy estimates 
for this population as well as analysis of changes across time. Results provide demographic and 
performance data for the prison population in comparison to the general U.S. adult population. 




Design and analysis methods ensure accurate comparisons across years 

In 1992 and 2003, the same sampling and data collection procedures were used to ensure that 
comparable populations were assessed in both years. The 2003 NAAL also used some of the same 
assessment tasks that were used in the 1992 NALS. About 45 percent of the tasks used in 2003 were 
drawn from those used in 1 992, while the remainder were newly created for the 2003 assessment. 
According to the widely used reference work Test Equating (Kolen and Brennan 1995, p. 248), using 
20 percent of the same items is sufficient to allow for comparisons between tests, provided that new 
items are developed following specifications similar to those used in developing the old items. The 
newly created NAAL tasks were modeled after the 1 992 tasks — having about the same average level 
of difficulty, requiring similar skills for successful completion, and covering the same content areas 
(home and family, health and safety, community and citizenship, consumer economics, work, and 
leisure and recreation). Item response theory (IRT;see,e.g., Baker 2001) was employed to link the 1992 
and 2003 scales using the tasks common to both years. Another step taken to ensure accurate 
comparisons across years was to recompute the 1992 scores using the 2003 analysis procedures, 
which differed in some respects from those originally used to analyze the 1992 data (e.g., the rules 
for dealing with missing data had been modified). In addition, results from both 1 992 and 2003 were 
reported using a newly developed set of performance levels (discussed below). 

Performance levels describe task performance for various score ranges 

For some purposes, it is useful to report average scores. For example, the average prose, document, 
or quantitative literacy score of one group (e.g., males) can be compared with that of another (e.g., 
females). Also, the average score of a particular group or of the entire population of U.S. adults 
residing in households and prisons in 1992 can be compared with the score in 2003. 

Another useful way to report results is by grouping adults with similar scores into a relatively small 
number of categories, generally referred to as performance levels. Reporting the percentages of adults 
scoring at various performance levels is somewhat analogous to reporting the percentages of 
students receiving various letter grades (e.g., an A or a B) on a test. Performance levels serve as a 
useful tool for identifying and characterizing the relative strengths and weaknesses of adults falling 
within various ranges of literacy ability. Breaking the adult population into these levels allows 
analysts, policymakers, and others to examine and discuss the typical performance and capabilities 
of specified proportions of the adult population. 

NCES originally used five'literacy levels" to report the 1992 results. In preparation for reporting on adult 
literacy performance in 2003, NCES asked the National Research Council (NRC) to evaluate the original 
1992 literacy levels and recommend a set of performance levels that could be used in reporting 2003 
results and also applied to 1992 results in order to make comparisons across years. In response to 
NCES's request, NRC established the Committee on Performance Levels for Adult Literacy. A preliminary 
report released by the committee in April 2005 (Flauser et al. 2005) examines the original 1992 literacy 
levels, outlines the newly developed performance levels, and details the methodology and rationale 
underlying the new levels.The committee's report discusses each step in the process of developing the 
new levels.The following brief discussion highlights only a few key points. 




New levels were developed in an open, public, and scientific way 

According to the Committee on Performance Levels for Adult Literacy (Hauser et al. 2005), the 
original 1992 literacy levels "were not meant to reflect policy-based judgments about 
expectations for adult literacy. That is, the procedures used to develop the assessment did not 
involve identifying the level of skills adults need in order to function adequately in society. When 
findings . . .were released, however, the . . . levels were interpreted and discussed as if they 
represented standards for the level of literacy adults should have," leading to "unsupported 
inferences." The committee concluded that "some of the more important details about the 
process for determining the 1 992 . . . levels were not specified" and that "a more open and public 
process combined with more explicit documentation would lead to better understanding of how 
the . . . levels were determined and what inferences could be based on them." 



While development of the 1992 literacy levels had begun with rating and sorting the assessment 
tasks according to cognitive complexity (see Kirsch et al. 2000 for details), development of the new 
performance levels involved initial specification of levels intended to correspond to policy-relevant 
categories of adults. The committee specified the levels after reviewing information about the 1992 
and 2003 assessments and asking stakeholders to identify the ways in which results would be used. 
The committee then created preliminary descriptions that characterized the literacy skills of adults 
at each performance level. (These preliminary descriptions of the levels were refined at various 
points in the development process.) 
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The next step was to determine the score ranges to be included in each level. After reviewing the 
literature about methods for determining score ranges, the committee decided to use the"bookmark" 
method. The method was implemented by holding two sessions with panels of"judges" consisting of 
adult literacy practitioners, officials with state offices of adult education, middle and high school 
teachers, and experts in industrial and organizational psychology. The judges received descriptions of 
the performance levels along with booklets of assessment tasks, arranged from easiest to hardest. 
Each booklet contained tasks from a single literacy area (prose, document, or quantitative). (For the 
first session, the booklets contained the tasks used in 1992;forthe second session, they contained the 
tasks used in 2003.) The judges' job was to place "bookmarks" in the booklets to identify the sets of 
tasks that adults at each level were "likely" to perform correctly. Following the recommendation of the 
designers of the bookmark method (Mitzel et al. 2001), "likely" was defined as 67 percent of the time 
(or, stated another way, two out of three times). 

For each task, IRT procedures were used to determine the score associated with a 67 percent 
probability of performing the task correctly. As noted in the committee's report (Hauser et al. 2005), a 
hallmark of IRT is the way it describes the relationship between the probability of a correct response 
and the scores on a proficiency scale. The committee established preliminary score ranges for the 
performance levels based on the scores corresponding to a 67 percent success rate on tasks that 
judges had included in each level (figure 6). 




Figure 6. Use of the bookmark method in developing the new performance levels 



Judges inserted "bookmarks"to 
divide the tasks at each level. 
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312 


316 


324 



277 


280 


291 


296 



For each task, item response theory (IRT) procedures were 
used to determine the score associated with a 67 percent 
probability of correct performance. 



'The National Research Council's Committee on Performance Levels for Adult Literacy originally called this 
level "Advanced," but the National Center for Education Statistics changed the title to "Proficient" in order 
to better convey how adults scoring at this level perform. 

NOTE: This figure is a simplified representation that is not intended to show actual numbers of tasks or 
actual scores associated with tasks. 

SOURCE: U.S. Department of Education, National Center for Education Statistics. Based on information 
about the bookmark method provided in Measuring Literacy: Performance Levels for Adults, Interim Report 
(Flauser et al. 2005). 



The committee's report discusses in detail the various "technical and nontechnical considerations" 
leading to the choice of a 67 percent success rate for developing the new performance levels. One 
reason is that the 80 percent success rate used by NCES to develop the 1 992 literacy levels was 
judged "overly stringent given the uses of the assessment results." In the committee's opinion, such 
a stringent criterion is needed when an assessment (e.g., a licensing examination) requires "a high 
degree of certainty that the individual has truly mastered the specific content or skills," but not 
when an assessment (e.g., NALS or NAAL) has low stakes, "that is, no scores are reported for 
individuals, and no decisions affecting an individual are based on the results." 

For each of the three literacy areas, the bookmark method generated "cut scores" that indicated the 
lowest score to be included in each performance level. For example, a cut score of 244 marked the 
lower boundary of the Basic level of quantitative literacy. To refine the bookmark-based cut scores, 
they were compared with the 1992 scores associated with selected background variables, including 
educational attainment.ln setting cut scores, thejudges had referred only to skill-based descriptions 
of the levels (e.g., "Is able to ..."), not to any information about background variables. However, the 
committee felt that an examination of background variables would be useful in evaluating the 
reasonableness of the resulting scores. The criterion for selecting the background variables was 
potential usefulness for distinguishing between performance levels. For example, the Basic level was 
intended to correspond to adults who are ready for GED preparation services, while the Below Basic 
level was intended to correspond to adults who are in need of basic adult literacy services (including 



services for adult English language learners). The following background variables were identified as 
relevant to distinguishing between these two levels: having some high school education (versus 
none at all) and reporting that one reads well (versus not well). 

The committee developed a set of rules and procedures for using the selected background variables 
to make adjustments to the bookmark-based cut scores. For the Basic level of prose and document 
literacy, the cut scores associated with the selected background variables were about the same as 
the bookmark-based scores, which therefore did not need to be adjusted. For the Intermediate level 
of prose and document literacy, the cut scores associated with the background variables were 
somewhat lower than the bookmark-based cut scores (although the differences were relatively 
small when considering only educational attainment instead of all the variables identified as 
relevant). The committee's rules and procedures resulted in minor downward adjustments to these 
bookmark-based cut scores. For the highest level of prose and document literacy and for all levels of 
quantitative literacy, the cut scores associated with the background variables were also lower than 
the bookmark-based scores, and the differences were greater. For example, analysis of 1992 scores by 
the background variables yielded a cut score of 207 as the lower boundary of the Basic level of 
quantitative literacy (compared with the bookmark-based cut score of 244). Since analysis by 
selected background variables was intended merely to complement the bookmark method, 
adjustments were relatively minor even in cases with relatively large differences between the 
bookmark-based scores and the scores associated with background variables. In this example, the 
final cut score for the Basic level of quantitative literacy was 235. Table 1 shows the final score ranges 
associated with the new performance levels. 



Table 1 . Score ranges associated with the new performance levels, by literacy area 





Below Basic 




Basic 


Intermediate 


Proficient 1 


Prose 


0-209 




210-264 


265-339 


340-500 


Document 


0-204 




205-249 


250-334 


335-500 


Quantitative 


0-234 




235-289 


290-349 


350-500 



’The National Research Council's Committee on Performance Levels for Adult Literacy originally called this 
level "Advanced," but the National Center for Education Statistics changed the title to "Proficient" in order to 
better convey how adults scoring at this level perform. 

SOURCE: Hauser, R.M., Edley, C.F., Jr., Koenig, J.A., and Elliot, S.W. (Eds.). (2005). Measuring Literacy: Performance 
Levels for Adults, Interim Report. National Research Council, Board on Testing and Assessment, Committee on 
Performance Levels for Adult Literacy. Washington, DC:The National Academies Press. 



The new levels are supplemented by a Nonliterate in English category 



As defined in the NRC 
committee's report (Hauser et al. 
2005), the Nonliterate in English 
category does not include the 
group of adults who could not be 
tested because they knew neither 
English nor Spanish. However, 
NCES decided to include this 
group in order to provide a more 
complete representation of the 
proportion ofU.S. adults who are 



In addition to the four performance levels that were developed using the bookmark method, the 
Committee on Performance Fevels for Adult Fiteracy also recommended that NCES report on a fifth 
category — Nonliterate in English. This category includes two groups of adults: 

■ Two percent of the adults who were selected to participate in the 2003 NAAF could not be 
tested — in other words, could not participate in NAAF at all — because they knew neither 
English nor Spanish (the other language spoken by interviewers in most areas). The Nonliterate 
in English category includes these adults because their inability to communicate in English 
indicates a lack of English literacy skills. 



not literate in English. 




■ Three percent of the adults who were tested in 2003 did not take the main part of the 
assessment, which was too difficult for them, but did take an alternative assessment specifically 
designed for the least-literate adults. Questions on the alternative assessment were asked in 
either English or Spanish, but all written materials were in English only. While some adults in this 
group displayed minimal English literacy skills (e.g., the ability to identify a letter or a common 
word in a simple text), others lacked such skills entirely. (For example, an adult who was able to 
attempt the alternative assessment by following oral Spanish instructions might still prove 
unable to do even the minimal amount of English reading needed to provide any correct 
answers.) The Nonliterate in English category includes these adults because their English literacy 
skills are minimal at best. 

In 2003, the two groups of adults classified as Nonliterate in English — the 2 percent who could not be 
tested because of a language barrier (i.e., inability to communicate in English or Spanish) and the 
3 percent who took the alternative assessment — accounted for 1 1 million adults, or 5 percent of the 
population. These adults range from having no English literacy skills to being able to"recognize some 
letters, numbers, or common sight words in everyday contexts" (Hauser et al. 2005). 

It is not possible to report on the Nonliterate in English category for 1992. This is because the 1992 
NALS did not include an alternative assessment for the least-literate adults, and an unknown 
proportion of the Below Basic population in that year is likely to have required such an assessment. 
(In 2003, adults were routed to the alternative assessment if they were unable to successfully 
perform a minimum number of easy literacy screening tasks. Although the 1992 assessment also 
began with a set of easy tasks, these tasks were different from the ones used in 2003. In 2003, 
moreover, questions for the screening tasks could be offered in Spanish, whereas only English was 
used in 1992.) 

As described above, the Nonliterate in English category includes all adults identified as lacking literacy 
in English — not only the lowest performers among adults who were able to participate in NAAL, but 
also adults who could not be tested because of a language barrier. In contrast, NAAL literacy 
results — reported in terms of scores and performance levels — provide information only about 
adults who could be tested. Figure 7 summarizes the composition of the Nonliterate in English 
category and its relationship to the NAAL literacy results. 

The reason that NAAL literacy results do not include adults who could not be tested as a result of a 
language barrier is that no performance data are available for these adults and it cannot be assumed 
that they would perform similarly to other adults with similar characteristics (e.g., age, gender, and 
education level). Such an assumption would be the basis of any approach designed to estimate their 
performance using a statistical model in the absence of data. While this assumption could be made 
for adults who failed to participate for reasons that do not relate to literacy (e.g., unavailability), it 
would overestimate the performance of adults who were untestable because of a language barrier. 
On the other hand, some of these untestable adults might have been able to perform a few of the 
easiest tasks correctly if, for example, the instructions had been given in their native language.There 
is no way to know for certain whether individual untestable adults would or would not have been 
able to provide a few correct answers. 



Figure 7. The Nonliterate in English category 



About 3 percent of adults in 
1 992 (vs. about 2 percent in 
2003) were considered to be 
untestable as a result of a 
language barrier. The fact that 
interviewers administering 
literacy screening tasks in 2003 
had the option of asking the 
questions in Spanish may be 
one reason that more adults 
could be tested in that year. 



Two groups 
of adults 



■ 2% of adults in 
2003. 

■ 3% of adults in 
1992. 


Unable to 
participate at all 
because of 
language barrier. 1 




■ 3% of adults in 
2003. 

■ Unknown 
percentage of 
adults in 1 992. 


Able to participate 
in alternative 
assessment for the 
least-literate 
adults. 



■ No scores can be 
estimated. 

■ Not included in 
NAAL literacy 
results. 



■ Scores can be 
estimated. 2 

■ Included in Below 
Basic performance 
level. 



'Adults in this group could communicate in neither English nor Spanish. (Although the assessment tasks 
measure literacy in English only, bilingual interviewers were available in most areas.) 

2 These adults' performance on a set of seven easy screening tasks is included when computing NAAL literacy 
results. (The screening tasks were used to determine which adults required the alternative assessment. For 
more information, see "The least-literate adults take an alternative assessment," in section 3.) 

NOTE: Adults are defined as people age 16 and older living in households or prisons. The Nonliterate in 
English category is reported only for 2003. This category is not reported for 1992 because there was no 
alternative assessment in 1 992 and an unknown proportion of the Below Basic population in that year is 
likely to have required such an assessment. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 1992 National Adult 
Literacy Survey (NALS) and 2003 National Assessment of Adult Literacy (NAAL). 



NCES adopted the new levels and refined their descriptions 

The new performance levels and related findings were presented to NCES as recommendations. 
Having accepted the general recommendations, NCES incorporated a few refinements before using 
the levels to report results. Table 2 presents descriptions and illustrative tasks selected by NCES to 
concisely convey the meaning of each level. More extensive descriptions of the levels appear in the 
report by the Committee on Performance Levels for Adult Literacy (Hauser et al. 2005). The 
committee provides a separate description of each level for each of the three literacy areas (prose, 
document, and quantitative). 

The new performance levels will be featured in publications that report on adult literacy 
performance in 2003 and differences in performance between 1 992 and 2003. Figure 8 shows the 
distribution of adults across the new levels in both years. More detailed comparisons of 
performance across years appear in A First Look at the Literacy of America's Adults in the 21st Century 
(Kutner, Greenberg, and Baer 2005). 





Table 2. Overview of the new performance levels 



Level and 

definition Key abilities associated with level Sample tasks typical of level 



Below Basic 

indicates no 
more than 
the most 
simple and 
concrete 
literacy skills. 


Adults at the Below Basic level' range from being 
nonliterate in English to having the abilities listed 
below: 

• locating easily identifiable information in short, 
commonplace prose texts 

• locating easily identifiable information and 
following written instructions in simple 
documents (e.g., charts or forms) 

• locating numbers and using them to perform 
simple quantitative operations (primarily 
addition) when the mathematical information is 
very concrete and familiar 


• searching a short, simple 
text to find out what a 
patient is allowed to drink 
before a medical test 

• signing a form 

• adding the amounts on a 
bank deposit slip 


Basic 


• reading and understanding information in short, 


• finding in a pamphlet for 


indicates 


commonplace prose texts 


prospective jurors an 


skills 




explanation of how people 


necessary to 




were selected for the jury 


perform 




pool 


simple and 
everyday 


• reading and understanding information in simple 


• using a television guide to 


documents 


find out what programs are 


literacy 

activities. 




on at a specific time 




• locating easily identifiable quantitative 


• comparing the ticket prices 




information and using it to solve simple, one-step 
problems when the arithmetic operation is 
specified or easily inferred 


for two events 


Intermediate 


• reading and understanding moderately dense, 


• consulting reference 


indicates 


less commonplace prose texts as well as 


materials to determine 


skills 


summarizing, making simple inferences, 


which foods contain a 


necessary to 


determining cause and effect, and recognizing 


particular vitamin 


perform 


the author's purpose 






• locating information in dense, complex 


• identifying a specific 


literacy 

activities. 


documents and making simple inferences 


location on a map 


about the information 






• locating less familiar quantitative information 


• calculating the total cost of 




and using it to solve problems when the 


ordering specific office 




arithmetic operation is not specified or easily 
inferred 


supplies from a catalog 


Proficient 


• reading lengthy, complex, abstract prose texts as 


• comparing viewpoints in 


indicates 


well as synthesizing information and making 


two editorials 


skills 


complex inferences 




necessary to 


• integrating, synthesizing, and analyzing multiple 


• interpreting a table about 


perform more 


pieces of information located in complex 


blood pressure, age, and 


complex and 

challenging 

literacy 


documents 


physical activity 


• locating more abstract quantitative information 


• computing and comparing 


activities. 


and using it to solve multistep problems when 


the cost per ounce of food 




the arithmetic operations are not easily inferred 
and the problems are more complex 


items 



To provide information about 
demographic, social, and 
economic factors associated with 
adults at the Below Basic level, 
NCES has profiled these adults in 
terms of various background 
characteristics (see Kutner, 
Greenberg, and Baer 2005). 

For each performance level, the 
percentage of adults who 
responded correctly to almost 
1 00 differen t assessment tasks is 
included in the Sample 
Questions section of the NAAL 
website (http://nces.ed.gov/ 
naal/TestOuestions.asD) . For a 
brief discussion and an example 
of how specific tasks relate to 
the levels, see "Each performance 
level represents a continuum of 
abilities, " later in this section. 



'This level includes the lowest performers among those who could be tested. Adults whom interviewers 
determined to be untestable as a result of a language barrier are not included in the literacy results at all, 
because no literacy scores could be estimated for them. For more information, see "The new levels are 
supplemented by a Nonliterate in English category," earlier in this section. 

NOTE: These performance levels are used to report results from the 2003 National Assessment of Adult 
Literacy (NAAL), including comparisons with results from the 1992 National Adult Literacy Survey (NALS). 
Although some of these performance levels share common names with levels used for the National 
Assessment of Educational Progress (NAEP), they do not correspond to the NAEP levels. 

SOURCE: U.S. Department of Education, National Center for Education Statistics (NCES). Based on 
information from Measuring Literacy: Performance Levels for Adults, Interim Report (Elauser et al. 2005) as well 
as input from the NCES Assessment Division. 




Figure 8. Percentage of adults scoring at each performance level, by literacy area: 1 992 
and 2003 



The percentage of adults at each 
level varies by literacy area (prose, 
document, or quantitative). 



Literacy area and year 
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Prose 

2003 
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33 
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‘Significantly different from 1992. 

NOTE: Adults are defined as people age 1 6 and older living in households or prisons. Adults who could not 
be interviewed due to either a language barrier or a cognitive or mental disability (3 percent in 2003 and 
4 percent in 1992) are excluded from this figure. (For more information, see "Administration procedures 
accommodate adults with special needs," in section 5.) Detail may not sum to totals because of rounding.The 
1992 results presented in this figure are based on reanalysis of the 1992 data using procedures developed 
for the 2003 assessment. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 1 992 National Adult Literacy 
Survey (NALS). Taken from figure 2 in A First Look at the Literacy of America's Adults in the 21 st Century (Kutner, 
Greenberg, and Baer 2005). 



Note that more adults scored at the lower levels for quantitative literacy than for prose and 
document literacy.One possible explanation for this is that NAAL quantitative tasks typically require 
most, if not all, of the skills typically required for prose or document tasks, plus specific quantitative 
skills. The skills common to the other task types are needed to effectively use the prose or 
document texts in which numbers for quantitative tasks are embedded, while the specific 
quantitative skills are needed to identify and perform the necessary computations. 

Each performance level represents a continuum of abilities 

Although certain tasks can be characterized as typical of each performance level (as shown in 
table 2, on the previous page), it is important to remember that the tasks at each level extend across 
a certain range of difficulty and therefore require a continuum of abilities. For exam pie, the new Basic 
level of document literacy encompasses scores ranging from 205 to 249. Adults with a score of 205 
(the lowest score included in the level) have a 67 percent rate of success with the easiest task at that 
level, while adults with a score of 249 (the highest score included in the level) have a 67 percent rate 
of success with the most difficult task at that level. This means that adults at the high end of the Basic 
level have an even higher rate of success with some of the level's easiest tasks. Moreover, these adults 
have a fairly high rate of success with some of the tasks at the low end of the Intermediate level, even 
though the rate is below 67 percent. 



Regardless of the specific criteria used to establish performance \eve\s,adults at every level have some 
probability of performing any task correctly. Therefore, it is not correct to say that adults at a certain 
performance level are "not able to do" tasks at higher levels. These adults are, however, less likely to 
succeed with such tasks. Figure 9 illustrates this fact by showing the percentage of adults at each of 
the new performance levels who were able to correctly perform a moderately difficult document 
task in 1992. For this particular task, the success rate ranged from 8 percent at the Below Basic level 
to 97 percent at the Proficient level. 



Figure 9. Percentage of adults able to correctly perform a sample document task in 
1992, by performance level and scale score 

Moderately difficult document task — Determine correct type of sandpaper from table 
Percent correct 




All Below Basic Basic Intermediate Proficient 
Adults (0-204) (205-249) (250-334) (335-500) 



Performance level for document literacy 



Percent correct 




Section 1 of this publication 
shows this sample task in its 
entirety (see figure 3). 

Although this task is associated 
with the Intermediate level, 
some adults at each level 
performed the task correctly. 
Adults with a score of 266 had a 
67 percent probability of correct 
performance. 



NOTE: Adults are defined as people age 1 6 and older living in households or prisons. The 4 percent of adults 
who could not be interviewed in 1992 due to either a language barrier ora cognitive or mental disability are 
excluded from this figure. (For more information, see "Administration procedures accommodate adults with 
special needs," in section 5.) In the line graph shown in this figure, the dotted vertical lines separate the 
performance levels.The solid guidelines draw attention to the relationship between a specific point on the 
scale — in this case, 266 — and a 67 percent probability of correct performance. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, 1992 National Adult Literacy 
Survey (NALS). 
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The 2003 National Assessment of Adult Literacy (NAAL) provides information about skills that are 
needed to successfully perform literacy tasks as well as skill deficits that impede task performance. 
Skills required for successful task performance range from basic, word-level skills (such as 
recognizing words) to higher level skills (such as drawing appropriate inferences from continuous 
text). Although different literacy tasks vary in the specific skills that they require, successful 
performance of virtually all literacy tasks requires at least some fundamental skills, such as the ability 
to read and understand common words. 

A variety of skills are needed to perform everyday literacy tasks 

Using printed and other written materials in everyday life requires multiple skills. The specific skills 
needed (as well as the necessary degree of skill proficiency) vary depending on the materials used 
and the task at hand. For example, computation skills are required only for quantitative tasks — such 
as determining how much paint to buy for a 20- by 30-foot room. While basic reading skills are 
needed for all literacy tasks, a higher level of these skills may be needed when the task requires 
understanding words that are less common and more difficult. Similarly, text search skills are used for 
finding a doctor's name and room number in a building directory as well as for locating a particular 
piece of information in a complex table, but the latter task requires a higher level of these skills. 

The difficulty of a particular task is determined by the specific actions required (also called the task 
demands) and the characteristics of the written materials used for the task. Some types of task 
demands are generally less challenging than others. For example, reading words is generally less 
challenging than making inferences based on the text that one has read. Of course, some words are 
easier to read than others, and some inferences are easier to make than others.The level of skill needed 
to meet a task demand depends both on the nature of the demand itself (e.g., locating specific words) 
and on related text characteristics (e.g., alphabetical vs. random order of a word list to be searched). In 
order to successfully perform a task, an individual must be able to apply each required skill at the 
required level. Flere is where each individual's unique skill development comes into play. For example, 
a task might involve locating several prices in a dense text and then comparing them. An individual 
with strong computation skills but weak text search skills might find it easier to compare the prices 
than to locate them.To accomplish the task, however, the individual would need to have adequate skills 
for meeting all the task demands. Underlying this analysis is the hypothesis that an individual's 
performance of a particular literacy task is jointly influenced by three key factors: the text characteristics, 
the task demands, and the individual's skills (figure 10). 



Figure 1 0. Factors influencing an individual's performance of a literacy task 




Whether an individual can 
successfully perform a particular 
literacy task depends on the 
characteristics of the text used 
for the task, the requirements or 
demands set by the task itself, 
and the skills of the individual 
performing the task. 



SOURCE: White, S. : and McCloskey, M. (forthcoming). Framework for the 2003 National Assessment of Adult 
Literacy (NCES 2006-473). U.S. Department of Education. Washington, DC: National Center for Education 
Statistics. 



NAAL includes new data on basic skills and the least-literate adults 

The 1992 National Adult Literacy Survey (NALS) was not able to provide much information about 
America's I east- 1 iterate adults, because these adults had trouble performing even the easiest 
assessment tasks. Thus, the 1992 NALS provided some indication of what these adults could nor do, 
but almost no information about what they could do. To address this problem, the 2003 NAAL 
includes a new alternative assessment (discussed below) that features easier tasks and texts and is 
administered only to adults whose skills are not adequate for meaningful participation in the "main 
NAAL" (i.e., the main part of the assessment). The 2003 NAAL also includes a new component 
(discussed later in this section) that uses oral reading tasks to measure the basic reading skills of all 
adults. Data provided by this component will help clarify the role that basic skills play in literacy task 
performance. 

The least-literate adults take an alternative assessment 

The least-literate adults' literacy skill levels and associated task performance are of particular interest 
to policymakers and literacy practitioners. The Adult Literacy Supplemental Assessment (ALSA) 
addresses the need for information about these adults, who would be able to perform few, if any, of 
the main NAAL assessment tasks. During the 2003 NAAL data collection, each participant first 
answered a background questionnaire and then was given a set of seven easy core screening tasks. 
Very low performance on the core screening tasks identified adults for whom the main NAAL would 
be too difficult and therefore would not provide a meaningful measure of performance. The 
screening process was designed to be conservative in routing adults to ALSA, so that all adults who 
were capable of meaningfully participating in the main NAAL would have an opportunity to do so. 
The adults who took ALSA (instead of the main NAAL) represented about 3 percent of all U.S. adults 
who were capable of being interviewed in English or Spanish (figure 1 1).The majority (60 percent) 
of ALSA participants were Hispanic. 



